{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NBO00BhA1aV6"
   },
   "source": [
    "Install the required dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Gfi2vdNiiHWT",
    "outputId": "2271a719-ac51-4228-94d0-23b0db093645"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.3/2.3 MB\u001b[0m \u001b[31m27.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.3/49.3 kB\u001b[0m \u001b[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h"
     ]
    }
   ],
   "source": [
    "!pip install lancedb langchain openai tiktoken pypdf tantivy sentence-transformers langchain-community -q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "je__kyDIyI62"
   },
   "source": [
    "# download your choice pdf file & upload here\n",
    "\n",
    " https://github.com/akashAD98/dummy_data/blob/main/microsoft_annual_report_2022.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "QcfZ_lr6FrEA"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "import lancedb\n",
    "import getpass\n",
    "import pandas as pd\n",
    "from lancedb.embeddings import EmbeddingFunctionRegistry\n",
    "from lancedb.pydantic import LanceModel, Vector\n",
    "from lancedb.embeddings import get_registry\n",
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "from langchain.document_loaders import PyPDFLoader\n",
    "from langchain.embeddings import OpenAIEmbeddings\n",
    "from langchain.embeddings import HuggingFaceEmbeddings\n",
    "from langchain_community.vectorstores import LanceDB\n",
    "\n",
    "\n",
    "# Set the OPENAI_API_KEY environment variable\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"sk-proj-....\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "NtbIb3QPjfuW"
   },
   "outputs": [],
   "source": [
    "def word_wrap(string, n_chars=72):\n",
    "    # Wrap a string at the next space after n_chars\n",
    "    if len(string) < n_chars:\n",
    "        return string\n",
    "    else:\n",
    "        return (\n",
    "            string[:n_chars].rsplit(\" \", 1)[0]\n",
    "            + \"\\n\"\n",
    "            + word_wrap(string[len(string[:n_chars].rsplit(\" \", 1)[0]) + 1 :], n_chars)\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7JSJy4AROGOM"
   },
   "source": [
    "We are using langchain fro loading the pdf file &  doing prprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0TUtpO0_jkUY",
    "outputId": "1caeb35f-6412-4861-dfac-a74218bddd20"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-6-393fdf8f6934>:9: LangChainDeprecationWarning: The class `OpenAIEmbeddings` was deprecated in LangChain 0.0.9 and will be removed in 1.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run `pip install -U langchain-openai` and import as `from langchain_openai import OpenAIEmbeddings`.\n",
      "  embeddings = OpenAIEmbeddings(model=\"text-embedding-3-large\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "35   \n",
      "Reportable Segments  \n",
      "Fiscal Year 2022 Compared with Fiscal Year 2021  \n",
      "Productivity and Business Processes  \n",
      "Revenue increased $9.4  billion or 18%.  \n",
      "• Office Commercial products and cloud services revenue increased $4.4  billion or 13%. Office 365 \n",
      "Commercial revenue grew 18% driven by seat growth of 14%, with continued momentum in small and \n",
      "medium business and frontline worker offerings, as well as growth in revenue per user. Office Commercial \n",
      "products revenue declined 22% driven by continued customer shift to cloud offerings.  \n",
      "• Office Consumer products and cloud services revenue increased $641  million or 11% driven by Microsoft 365 \n",
      "Consumer subscription revenue. Microsoft 365 Consumer subscribers grew 15% to 59.7  million.  \n",
      "• LinkedIn revenue increased $3.5  billion or 34% driven by a strong job market in our Talent Solutions business \n",
      "and advertising demand in our Marketing Solutions business.  \n",
      "• Dynamics products and cloud services revenue increased 25% driven by Dynamics 365 growth of 39%.  \n",
      "Operating income increased $5.3  billion or 22%.  \n",
      "• Gross margin increased $7.3  billion or 17% driven by growth in Office 365 Commercial and LinkedIn. Gross \n",
      "margin percentage was relatively unchanged. Excluding the impact of the change in accounting estimate, \n",
      "gross margin percentage increased 2 points driven by improvement across all cloud services.  \n",
      "• Operating expenses increased $2.0  billion or 11% driven by investments in LinkedIn and cloud engineering.  \n",
      "Gross margin and operating income both included an unfavorable foreign currency impact of 2%.  \n",
      "Intelligent Cloud  \n",
      "Revenue increased $15.2  billion or 25%.  \n",
      "• Server products and cloud services revenue increased $14.7  billion or 28% driven by Azure and other cloud \n",
      "services. Azure and other cloud services revenue grew 45% driven by growth in our consumption -based \n",
      "services. Server products revenue increased 5% driven by hybrid solutions, including Windows Server and \n",
      "SQL Server running in multi -cloud environments.  \n",
      "• Enterprise Services revenue increased $464  million or 7% driven by growth in Enterprise Support Services.  \n",
      "Operating income increased $6.6  billion or 25%.  \n",
      "• Gross margin increased $9.4  billion or 22% driven by growth in Azure and other cloud services. Gross margin \n",
      "percentage decreased. Excluding the impact of the change in accounting estimate, gross margin percentage \n",
      "was relatively unchanged driven by improvement in Azure and other cloud services, offset in part by sales \n",
      "mix shift to Azure and other cloud services.  \n",
      "• Operating expenses increased $2.8  billion or 16% driven by investments in Azure and other cloud services.  \n",
      "Revenue and operating income included an unfavorable foreign currency impact of 2% and 3%, respectively.  \n",
      "More Personal Computing  \n",
      "Revenue increased $5.6  billion or 10%.  \n",
      "• Windows revenue increased $2.3  billion or 10% driven by growth in Windows OEM and Windows \n",
      "Commercial. Windows OEM revenue increased 11% driven by continued strength in the commercial PC \n",
      "market, which has higher revenue per license. Windows Commercial products and cloud services revenue \n",
      "increased 11% driven by demand for Microsoft 365.\n"
     ]
    }
   ],
   "source": [
    "# your pdf file\n",
    "loader = PyPDFLoader(\"/content/microsoft_annual_report_2022.pdf\")\n",
    "documents = loader.load_and_split()\n",
    "\n",
    "# split it into chunks\n",
    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "docsn = text_splitter.split_documents(documents)\n",
    "\n",
    "embeddings = OpenAIEmbeddings(model=\"text-embedding-3-large\")\n",
    "\n",
    "docsearch = LanceDB.from_documents(docsn, embeddings)\n",
    "\n",
    "query = \"What were the most important factors that contributed to increases in revenue?\"\n",
    "docsnew = docsearch.similarity_search(query)\n",
    "print(docsnew[0].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nr7_uISgfHo4"
   },
   "source": [
    "### Query expansion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "B7B0XA3ajsFY"
   },
   "outputs": [],
   "source": [
    "# we are using openai for generating query\n",
    "import os\n",
    "import openai\n",
    "from openai import OpenAI\n",
    "\n",
    "openai.api_key = os.environ[\"OPENAI_API_KEY\"]\n",
    "openai_client = OpenAI()\n",
    "\n",
    "\n",
    "def augment_query_generated(query, model=\"gpt-3.5-turbo\"):\n",
    "    messages = [\n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": \"You are a helpful expert financial research assistant. Provide an example answer to the given question, that might be found in a document like an annual report. \",\n",
    "        },\n",
    "        {\"role\": \"user\", \"content\": query},\n",
    "    ]\n",
    "\n",
    "    response = openai_client.chat.completions.create(\n",
    "        model=model,\n",
    "        messages=messages,\n",
    "    )\n",
    "    content = response.choices[0].message.content\n",
    "    return content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-2v8mN9XQy7Y"
   },
   "source": [
    "### its generating  query & hypothetica answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "IPBqpky3jtwM",
    "outputId": "5ee48a12-f633-4bc2-c0a1-af47cb3851b7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What were the most important factors that contributed to increases in\n",
      "revenue? In the current fiscal year, our company experienced\n",
      "significant revenue growth due to a combination of factors. Firstly, we\n",
      "successfully launched several new products that were well-received in\n",
      "the market, leading to increased sales and market share. Additionally,\n",
      "our strategic marketing initiatives helped to enhance brand awareness\n",
      "and attract new customers. Furthermore, we expanded our distribution\n",
      "network, allowing us to reach a wider customer base and drive higher\n",
      "sales volume. Lastly, cost control measures and operational\n",
      "efficiencies enabled us to improve profit margins and maximize revenue\n",
      "generation. Overall, the successful execution of these strategies\n",
      "significantly contributed to the increase in revenue during the year.\n"
     ]
    }
   ],
   "source": [
    "original_query = (\n",
    "    \"What were the most important factors that contributed to increases in revenue?\"\n",
    ")\n",
    "hypothetical_answer = augment_query_generated(original_query)\n",
    "# we are combining our orignal query + hypothetical_answer\n",
    "joint_query = f\"{original_query} {hypothetical_answer}\"\n",
    "print(word_wrap(joint_query))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "daOypkNqjwIh",
    "outputId": "ed6ae767-f0fd-4382-e5d5-138c97147a22"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Document(metadata={'page': 36, 'source': '/content/microsoft_annual_report_2022.pdf'}, page_content='35   \\nReportable Segments  \\nFiscal Year 2022 Compared with Fiscal Year 2021  \\nProductivity and Business Processes  \\nRevenue increased $9.4  billion or 18%.  \\n• Office Commercial products and cloud services revenue increased $4.4  billion or 13%. Office 365 \\nCommercial revenue grew 18% driven by seat growth of 14%, with continued momentum in small and \\nmedium business and frontline worker offerings, as well as growth in revenue per user. Office Commercial \\nproducts revenue declined 22% driven by continued customer shift to cloud offerings.  \\n• Office Consumer products and cloud services revenue increased $641  million or 11% driven by Microsoft 365 \\nConsumer subscription revenue. Microsoft 365 Consumer subscribers grew 15% to 59.7  million.  \\n• LinkedIn revenue increased $3.5  billion or 34% driven by a strong job market in our Talent Solutions business \\nand advertising demand in our Marketing Solutions business.  \\n• Dynamics products and cloud services revenue increased 25% driven by Dynamics 365 growth of 39%.  \\nOperating income increased $5.3  billion or 22%.  \\n• Gross margin increased $7.3  billion or 17% driven by growth in Office 365 Commercial and LinkedIn. Gross \\nmargin percentage was relatively unchanged. Excluding the impact of the change in accounting estimate, \\ngross margin percentage increased 2 points driven by improvement across all cloud services.  \\n• Operating expenses increased $2.0  billion or 11% driven by investments in LinkedIn and cloud engineering.  \\nGross margin and operating income both included an unfavorable foreign currency impact of 2%.  \\nIntelligent Cloud  \\nRevenue increased $15.2  billion or 25%.  \\n• Server products and cloud services revenue increased $14.7  billion or 28% driven by Azure and other cloud \\nservices. Azure and other cloud services revenue grew 45% driven by growth in our consumption -based \\nservices. Server products revenue increased 5% driven by hybrid solutions, including Windows Server and \\nSQL Server running in multi -cloud environments.  \\n• Enterprise Services revenue increased $464  million or 7% driven by growth in Enterprise Support Services.  \\nOperating income increased $6.6  billion or 25%.  \\n• Gross margin increased $9.4  billion or 22% driven by growth in Azure and other cloud services. Gross margin \\npercentage decreased. Excluding the impact of the change in accounting estimate, gross margin percentage \\nwas relatively unchanged driven by improvement in Azure and other cloud services, offset in part by sales \\nmix shift to Azure and other cloud services.  \\n• Operating expenses increased $2.8  billion or 16% driven by investments in Azure and other cloud services.  \\nRevenue and operating income included an unfavorable foreign currency impact of 2% and 3%, respectively.  \\nMore Personal Computing  \\nRevenue increased $5.6  billion or 10%.  \\n• Windows revenue increased $2.3  billion or 10% driven by growth in Windows OEM and Windows \\nCommercial. Windows OEM revenue increased 11% driven by continued strength in the commercial PC \\nmarket, which has higher revenue per license. Windows Commercial products and cloud services revenue \\nincreased 11% driven by demand for Microsoft 365.'), Document(metadata={'page': 32, 'source': '/content/microsoft_annual_report_2022.pdf'}, page_content='31   \\nIndustry Trends  \\nOur industry is dynamic and highly competitive, with frequent changes in both technologies and business models. Each \\nindustry shift is an opportunity to conceive new products, new technologies, or new ideas that can further transform the \\nindustry and our business. At Microsoft, we push the boundaries of what is possible through a broad range of research \\nand development activities that seek to identify and address the changing demands of customers and users, industry \\ntrends, and competitive forces.  \\nEconomic Conditions, Challenges, and Risks  \\nThe markets for software, devices, and cloud -based services are dynamic and highly competitive. Our competitors are \\ndeveloping new software and devices, while also deploying competing cloud -based services for consumers and \\nbusinesses. The devices and form factors customers prefer evolve rapidly, and influence how users access services in \\nthe cloud, and in some cases, the user’s choice of which suite of cloud -based services to use. We must continue to evolve \\nand adapt over an extended time in pace with this changing environment. The investments we are making in infrastructure \\nand devices will continue to increase our operating costs and may decrease our operating margins.  \\nOur success is highly dependent on our ability to attract and retain qualified employees. We hire a mix of university and \\nindustry talent worldwide. We compete for talented individuals globally by offering an exceptional working environment, \\nbroad customer reach, scale in resources, the ability to grow one’s career across many different products and businesses, \\nand competitive compensation and benefits. Aggregate demand for our software, services, and devices is correlated to \\nglobal macroeconomic and geopolitical factors, which remain dynamic.  \\nOur devices are primarily manufactured by third -party contract manufacturers, some of which contain certain components \\nfor which there are very few qualified suppliers. For these components, we have limited near -term flexibility to use other \\nmanufacturers if a current vendor becomes unavailable or is unable to meet our requirements. Extended disruptions at \\nthese suppliers and/or manufacturers could lead to a similar disruption in our ability to manufacture devices on time to \\nmeet consumer demand.  \\nOur international operations provide a significant portion of our total revenue and expenses. Many of these revenue and \\nexpenses are denominated in currencies other than the U.S. dollar. As a result, changes in foreign exchange rates may \\nsignificantly affect revenue and expenses. Fluctuations in the U.S. dollar relative to certain foreign currencies did not hav e \\na material impact on reported revenue or expenses from our international operations in fiscal year 2022.  \\nRefer to Risk Factors in our fiscal year 2022 Form 10 -K for a discussion of these factors and other risks.  \\nSeasonality  \\nOur revenue fluctuates quarterly and is generally higher in the second and fourth quarters of our fiscal year. Second \\nquarter revenue is driven by corporate year -end spending trends in our major markets and holiday season spending by \\nconsumers, and fourth quarter revenue is driven by the volume of multi -year on -premises contracts executed during the \\nperiod.  \\nReportable Segments  \\nWe report our financial performance based on the following segments: Productivity and Business Processes, Intelligent \\nCloud, and More Personal Computing. The segment amounts included in MD&A are presented on a basis consistent with \\nour internal management reporting. Additional information on our reportable segments is contained in Note 19 – Segment \\nInformation and Geographic Data of the Notes to Financial Statements in our fiscal year 2022 Form 10 -K.  \\nMetrics  \\nWe use metrics in assessing the performance of our business and to make informed decisions regarding the allocation of'), Document(metadata={'page': 34, 'source': '/content/microsoft_annual_report_2022.pdf'}, page_content='33 Dynamics products and cloud services revenue growth  Revenue from Dynamics products and cloud services, \\nincluding Dynamics 365, comprising a set of intelligent, \\ncloud -based applications across ERP, CRM, Customer \\nInsights, Power Apps, and Power Automate; and on -\\npremises ERP and CRM applications  \\n    \\nLinkedIn revenue growth  Revenue from LinkedIn, including Talent Solutions, \\nMarketing Solutions, Premium Subscriptions, and Sales \\nSolutions  \\n    \\nServer products and cloud services revenue growth  Revenue from Server products and cloud services, \\nincluding Azure and other cloud services; SQL Server, \\nWindows Server, Visual Studio, System Center, and \\nrelated Client Access Licenses (“CALs”); and Nuance and \\nGitHub  \\nMore Personal Computing  \\nMetrics related to our More Personal Computing segment assess the performance of key lines of business within this \\nsegment. These metrics provide strategic product insights which allow us to assess the performance across our \\ncommercial and consumer businesses. As we have diversity of target audiences and sales motions within the Windows \\nbusiness, we monitor metrics that are reflective of those varying motions.  \\n  \\nWindows OEM revenue growth  Revenue from sales of Windows Pro and non -Pro licenses sold \\nthrough the OEM channel  \\n    \\nWindows Commercial products and cloud \\nservices revenue growth  Revenue from Windows Commercial products and cloud services, \\ncomprising volume licensing of the Windows operating system, \\nWindows cloud services, and other Windows commercial offerings  \\n    \\nSurface revenue growth  Revenue from Surface devices and accessories  \\n    \\nXbox content and services revenue growth  Revenue from Xbox content and services, comprising first - and third -\\nparty content (including games and in -game content), Xbox Game \\nPass and other subscriptions, Xbox Cloud Gaming, third -party disc \\nroyalties, advertising, and other cloud services  \\n    \\nSearch and news advertising revenue, \\nexcluding TAC, growth  Revenue from search and news advertising excluding traffic \\nacquisition costs (“TAC”) paid to Bing Ads network publishers and \\nnews partners  \\nSUMMARY RESULTS OF OPERATIONS  \\n  \\n(In millions,  except  percentages  and per share  amounts)  2022  2021  Percentage  \\nChange    \\n        \\nRevenue  $  198,270   $  168,088    18%  \\nGross margin   135,620    115,856    17%  \\nOperating income   83,383    69,916    19%  \\nNet income   72,738    61,271    19%  \\nDiluted earnings per share   9.65   8.05   20%  \\n        \\nAdjusted net income (non -GAAP)   69,447    60,651    15%  \\nAdjusted diluted earnings per share (non -GAAP)   9.21   7.97   16%    \\nAdjusted net income and adjusted diluted earnings per share (“EPS”) are non -GAAP financial measures which exclude \\nthe net  income tax benefit related to transfer of intangible properties in the first quarter of fiscal year 2022 and the'), Document(metadata={'page': 35, 'source': '/content/microsoft_annual_report_2022.pdf'}, page_content='34 net income tax benefit related to an India Supreme Court decision on withholding taxes in the third quarter of fiscal year \\n2021. Refer to the Non -GAAP Financial Measures section below for a reconciliation of our financial results reported in \\naccordance with GAAP to non -GAAP financial results. See Note 12 – Income Taxes of the Notes to Financial Statements \\nin our fiscal year 2022 Form 10 -K for further discussion.  \\nFiscal Year 2022 Compared with Fiscal Year 2021  \\nRevenue increased $30.2  billion or 18% driven by growth across each of our segments. Intelligent Cloud revenue \\nincreased driven by Azure and other cloud services. Productivity and Business Processes revenue increased driven by \\nOffice 365 Commercial and LinkedIn. More Personal Computing revenue increased driven by Search and news \\nadvertising and Windows.  \\nCost of revenue increased $10.4  billion or 20% driven by growth in Microsoft Cloud.  \\nGross margin increased $19.8  billion or 17% driven by growth across each of our segments.  \\n• Gross margin percentage decreased slightly. Excluding the impact of the fiscal year 2021 change in \\naccounting estimate for the useful lives of our server and network equipment, gross margin percentage \\nincreased 1 point driven by improvement in Productivity and Business Processes.  \\n• Microsoft Cloud gross margin percentage decreased slightly to 70%. Excluding the impact of the change in \\naccounting estimate, Microsoft Cloud gross margin percentage increased 3 points driven by improvement \\nacross our cloud services, offset in part by sales mix shift to Azure and other cloud services.  \\nOperating expenses increased $6.3  billion or 14% driven by investments in cloud engineering, LinkedIn, Gaming, and \\ncommercial sales.  \\nKey changes in operating expenses were:  \\n• Research and development expenses increased $3.8  billion or 18% driven by investments in cloud \\nengineering, Gaming, and LinkedIn.  \\n• Sales and marketing expenses increased $1.7  billion or 8% driven by investments in commercial sales and \\nLinkedIn. Sales and marketing included a favorable foreign currency impact of 2%.  \\n• General and administrative expenses increased $793  million or 16% driven by investments in corporate \\nfunctions.  \\nOperating income increased $13.5  billion or 19% driven by growth across each of our segments.  \\nCurrent year net income and diluted EPS were positively impacted by the net tax benefit related to the transfer of \\nintangible properties, which resulted in an increase to net income and diluted EPS of $3.3  billion and $0.44, respectively. \\nPrior year net income and diluted EPS were positively impacted by the net tax benefit related to the India Supreme Court \\ndecision on withholding taxes, which resulted in an increase to net income and diluted EPS of $620  million and $0.08, \\nrespectively.  \\nGross margin and operating income both included an unfavorable foreign currency impact of 2%.  \\nSEGMENT RESULTS OF OPERATIONS  \\n  \\n(In millions, except percentages)  2022  2021  Percentage  \\nChange    \\n        \\nRevenue        \\n        \\nProductivity and Business Processes  $ 63,364   $ 53,915    18%  \\nIntelligent Cloud   75,251    60,080    25%  \\nMore Personal Computing   59,655    54,093    10%        \\nTotal  $   198,270   $  168,088    18%          \\n        \\nOperating Income        \\n        \\nProductivity and Business Processes  $ 29,687   $ 24,351    22%  \\nIntelligent Cloud   32,721    26,126    25%  \\nMore Personal Computing   20,975    19,439    8%        \\nTotal  $ 83,383   $ 69,916    19%')]\n"
     ]
    }
   ],
   "source": [
    "# we are passing the  joint_query & fining the  similar docs for this query\n",
    "retrieved_documents = docsearch.similarity_search(joint_query, k=4)\n",
    "print(retrieved_documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "C3InbcrAjyWE",
    "outputId": "b71aea80-861c-4d5b-c036-51aa0e64df33"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "35   \n",
      "Reportable Segments  \n",
      "Fiscal Year 2022 Compared with Fiscal Year 2021  \n",
      "Productivity and Business Processes  \n",
      "Revenue increased $9.4  billion or 18%.  \n",
      "• Office Commercial products and cloud services revenue increased $4.4  billion or 13%. Office 365 \n",
      "Commercial revenue grew 18% driven by seat growth of 14%, with continued momentum in small and \n",
      "medium business and frontline worker offerings, as well as growth in revenue per user. Office Commercial \n",
      "products revenue declined 22% driven by continued customer shift to cloud offerings.  \n",
      "• Office Consumer products and cloud services revenue increased $641  million or 11% driven by Microsoft 365 \n",
      "Consumer subscription revenue. Microsoft 365 Consumer subscribers grew 15% to 59.7  million.  \n",
      "• LinkedIn revenue increased $3.5  billion or 34% driven by a strong job market in our Talent Solutions business \n",
      "and advertising demand in our Marketing Solutions business.  \n",
      "• Dynamics products and cloud services revenue increased 25% driven by Dynamics 365 growth of 39%.  \n",
      "Operating income increased $5.3  billion or 22%.  \n",
      "• Gross margin increased $7.3  billion or 17% driven by growth in Office 365 Commercial and LinkedIn. Gross \n",
      "margin percentage was relatively unchanged. Excluding the impact of the change in accounting estimate, \n",
      "gross margin percentage increased 2 points driven by improvement across all cloud services.  \n",
      "• Operating expenses increased $2.0  billion or 11% driven by investments in LinkedIn and cloud engineering.  \n",
      "Gross margin and operating income both included an unfavorable foreign currency impact of 2%.  \n",
      "Intelligent Cloud  \n",
      "Revenue increased $15.2  billion or 25%.  \n",
      "• Server products and cloud services revenue increased $14.7  billion or 28% driven by Azure and other cloud \n",
      "services. Azure and other cloud services revenue grew 45% driven by growth in our consumption -based \n",
      "services. Server products revenue increased 5% driven by hybrid solutions, including Windows Server and \n",
      "SQL Server running in multi -cloud environments.  \n",
      "• Enterprise Services revenue increased $464  million or 7% driven by growth in Enterprise Support Services.  \n",
      "Operating income increased $6.6  billion or 25%.  \n",
      "• Gross margin increased $9.4  billion or 22% driven by growth in Azure and other cloud services. Gross margin \n",
      "percentage decreased. Excluding the impact of the change in accounting estimate, gross margin percentage \n",
      "was relatively unchanged driven by improvement in Azure and other cloud services, offset in part by sales \n",
      "mix shift to Azure and other cloud services.  \n",
      "• Operating expenses increased $2.8  billion or 16% driven by investments in Azure and other cloud services.  \n",
      "Revenue and operating income included an unfavorable foreign currency impact of 2% and 3%, respectively.  \n",
      "More Personal Computing  \n",
      "Revenue increased $5.6  billion or 10%.  \n",
      "• Windows revenue increased $2.3  billion or 10% driven by growth in Windows OEM and Windows \n",
      "Commercial. Windows OEM revenue increased 11% driven by continued strength in the commercial PC \n",
      "market, which has higher revenue per license. Windows Commercial products and cloud services revenue \n",
      "increased 11% driven by demand for Microsoft 365.\n"
     ]
    }
   ],
   "source": [
    "# sample of retrieved_documents\n",
    "print(retrieved_documents[0].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "slsNEwLV-NWS"
   },
   "source": [
    "#Reranker model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nRDEYibze8xA"
   },
   "source": [
    "we are using landb for colbertv2 implementation\n",
    "COlbertv2 is reranker method helpful for improving the ranking of results its returning top ranking\n",
    "results as the most relevant to your particular query.\n",
    "\n",
    "for more technical details please check the blog :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZHjbtdCpkB-r",
    "outputId": "1f08c28b-466f-4b58-ffd8-b775175aca03"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "31   \n",
      "Industry Trends  \n",
      "Our industry is dynamic and highly competitive, with frequent changes in both technologies and business models. Each \n",
      "industry shift is an opportunity to conceive new products, new technologies, or new ideas that can further transform the \n",
      "industry and our business. At Microsoft, we push the boundaries of what is possible through a broad range of research \n",
      "and development activities that seek to identify and address the changing demands of customers and users, industry \n",
      "trends, and competitive forces.  \n",
      "Economic Conditions, Challenges, and Risks  \n",
      "The markets for software, devices, and cloud -based services are dynamic and highly competitive. Our competitors are \n",
      "developing new software and devices, while also deploying competing cloud -based services for consumers and \n",
      "businesses. The devices and form factors customers prefer evolve rapidly, and influence how users access services in \n",
      "the cloud, and in some cases, the user’s choice of which suite of cloud -based services to use. We must continue to evolve \n",
      "and adapt over an extended time in pace with this changing environment. The investments we are making in infrastructure \n",
      "and devices will continue to increase our operating costs and may decrease our operating margins.  \n",
      "Our success is highly dependent on our ability to attract and retain qualified employees. We hire a mix of university and \n",
      "industry talent worldwide. We compete for talented individuals globally by offering an exceptional working environment, \n",
      "broad customer reach, scale in resources, the ability to grow one’s career across many different products and businesses, \n",
      "and competitive compensation and benefits. Aggregate demand for our software, services, and devices is correlated to \n",
      "global macroeconomic and geopolitical factors, which remain dynamic.  \n",
      "Our devices are primarily manufactured by third -party contract manufacturers, some of which contain certain components \n",
      "for which there are very few qualified suppliers. For these components, we have limited near -term flexibility to use other \n",
      "manufacturers if a current vendor becomes unavailable or is unable to meet our requirements. Extended disruptions at \n",
      "these suppliers and/or manufacturers could lead to a similar disruption in our ability to manufacture devices on time to \n",
      "meet consumer demand.  \n",
      "Our international operations provide a significant portion of our total revenue and expenses. Many of these revenue and \n",
      "expenses are denominated in currencies other than the U.S. dollar. As a result, changes in foreign exchange rates may \n",
      "significantly affect revenue and expenses. Fluctuations in the U.S. dollar relative to certain foreign currencies did not hav e \n",
      "a material impact on reported revenue or expenses from our international operations in fiscal year 2022.  \n",
      "Refer to Risk Factors in our fiscal year 2022 Form 10 -K for a discussion of these factors and other risks.  \n",
      "Seasonality  \n",
      "Our revenue fluctuates quarterly and is generally higher in the second and fourth quarters of our fiscal year. Second \n",
      "quarter revenue is driven by corporate year -end spending trends in our major markets and holiday season spending by \n",
      "consumers, and fourth quarter revenue is driven by the volume of multi -year on -premises contracts executed during the \n",
      "period.  \n",
      "Reportable Segments  \n",
      "We report our financial performance based on the following segments: Productivity and Business Processes, Intelligent \n",
      "Cloud, and More Personal Computing. The segment amounts included in MD&A are presented on a basis consistent with \n",
      "our internal management reporting. Additional information on our reportable segments is contained in Note 19 – Segment \n",
      "Information and Geographic Data of the Notes to Financial Statements in our fiscal year 2022 Form 10 -K.  \n",
      "Metrics  \n",
      "We use metrics in assessing the performance of our business and to make informed decisions regarding the allocation of\n"
     ]
    }
   ],
   "source": [
    "db = lancedb.connect(\"/tmp/db\")\n",
    "registry = EmbeddingFunctionRegistry.get_instance()\n",
    "func = registry.get(\"openai\").create()\n",
    "\n",
    "\n",
    "class Words(LanceModel):\n",
    "    text: str = func.SourceField()\n",
    "    vector: Vector(func.ndims()) = func.VectorField()\n",
    "\n",
    "\n",
    "table = db.create_table(\"wordsllm\", schema=Words, mode=\"overwrite\")\n",
    "\n",
    "# data from retriver\n",
    "formatted_data = [{\"text\": doc.page_content} for doc in retrieved_documents]\n",
    "\n",
    "\n",
    "# ingest docs with auto-vectorization\n",
    "table.add(formatted_data)\n",
    "# Create the FTS index on the 'text' field\n",
    "table.create_fts_index([\"text\"], replace=True)\n",
    "\n",
    "\n",
    "# normal search\n",
    "query = \"technologies and business models\"\n",
    "actual = table.search(query).limit(1).to_pydantic(Words)[0]\n",
    "print(actual.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DWriE9uWMgPb"
   },
   "source": [
    "#Colbertv2 reranker model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "veDgcnjXxpIn",
    "outputId": "99bb825e-67a8-4e26-ddae-c78ad68ea7f3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting rerankers\n",
      "  Downloading rerankers-0.5.3-py3-none-any.whl.metadata (27 kB)\n",
      "Requirement already satisfied: pydantic in /usr/local/lib/python3.10/dist-packages (from rerankers) (2.9.1)\n",
      "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from rerankers) (4.66.5)\n",
      "Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic->rerankers) (0.7.0)\n",
      "Requirement already satisfied: pydantic-core==2.23.3 in /usr/local/lib/python3.10/dist-packages (from pydantic->rerankers) (2.23.3)\n",
      "Requirement already satisfied: typing-extensions>=4.6.1 in /usr/local/lib/python3.10/dist-packages (from pydantic->rerankers) (4.12.2)\n",
      "Downloading rerankers-0.5.3-py3-none-any.whl (37 kB)\n",
      "Installing collected packages: rerankers\n",
      "Successfully installed rerankers-0.5.3\n"
     ]
    }
   ],
   "source": [
    "!pip install rerankers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 521,
     "referenced_widgets": [
      "a3d72b89c337400da5899d07bdc2d766",
      "f9d863fbf47f41b88977cdd88690450d",
      "134497ffd3e94a5c9aa66e7c3203abd6",
      "9f4e1e842c7d43a5ab48355e89b636db",
      "4bdaca8ab0154560b86d3908ae3a66d6",
      "664c4e37bd94479b8b5a846955882907",
      "6622dfaaa7d54feaaae0c304f2d47d53",
      "2529aefb68a24dfda867d5cf2fb2c72d",
      "bdaf3aca7a854438b16b62813114e844",
      "5560c59190504890a52db90a38e827c6",
      "39b2f1fef1064793bed1df03a7f38dc0",
      "ff434191648d4962b1dd9d537b13e5b9",
      "2b84894caa1748f2bcd45d58d7daf10d",
      "b234b6274d1b4ad595a646bcc034cf0e",
      "268a9da7c00c4c95afd600c89635025d",
      "6ce5975d69ef4ac09a36ba3bf5eb6ea4",
      "6c8883cdd6e745d5825ee9a93cb4fcd9",
      "4ead8521f611425baf7cda79ff1ca050",
      "24de3fd8af3e4a5487a01988907a87e0",
      "a5dc7bb09f554c6c8c20b4eaec0d23c2",
      "09c87aa813dc4c63b8fbd269c16212bf",
      "cb7ecd9e3d844f749104d6ddea0ecd0d",
      "8bb5cc8a235d4b628a02fd5b8121c62c",
      "c9b353b7d8b04a9ba7f50c1a2aec444c",
      "4760d03b336d46af8ec56708363f0dae",
      "6ed4e269944e4199a65182a62cb21cf1",
      "332258cba6dd4f5bac2c01cfdc94da64",
      "39e17d13567f45d5a6eaa0489870942e",
      "54ee840aa7cc4e88b913670835806ad9",
      "61b8f37d881e4d9e992c9eb47b6da45a",
      "14eefbfcb6af409585a0ae98c8103124",
      "026b9c44ea90466abf7ec2712b833624",
      "434f9fc5b049407caa1bd10e2f119c20",
      "af979629e4b84ebaa32adf0d5395e2af",
      "1c1adb96821044488d3b62c6995e5abb",
      "fe86ad6e7d5444c89b52d30f73a7a4c2",
      "f057247bfb3b46beb3b72e5b922ee753",
      "6c7c1cec1e2449bfb89b30190ef4ecd3",
      "e5d0c642966140b5b23d517d338f28d3",
      "cdf5f7262b1a4f68a2b6176a16a98766",
      "13a077eee8cb4630b69080ceb9a72591",
      "95e0a7269ef04dc6833d340edfb33449",
      "60b52ad5a99f4f7e87c15848c9e89959",
      "96784fc1a5d54b0ab46760e2a0872cf5",
      "6c8dc5cadb674a33b37411aa1695bc7c",
      "a7fbe371939940b9b92a1be42f46f004",
      "52ff49f476324442b838d555c0afa4b0",
      "38261d426c014258af529fd8431656ab",
      "fd94250b445e4258883a5345005cdfcb",
      "a994fa9bc0184da3ae40c02481034853",
      "722c11cf2c6d4b58a55a06423028d254",
      "8b23323756914edf8737d83759d7602a",
      "6ea1cf9ed0544a58a4d6c3bc29149d50",
      "f92393e54d32481095e2bb4079857da4",
      "69f05dce78db46d0a610d04cad2001c4",
      "0562b3dc436c44a8b6a02e45936f2435",
      "c837aa7e9e8f436d839b11a44d6cea79",
      "791bfc04d4c741209a53f6d7d73a45a6",
      "856f2919948044f29f482b1fc7989fba",
      "05d1933333b94afc835d13f92b70dc29",
      "ccefa430d75249aa81aa205ca180fec5",
      "d7706a812ddb4580bb07da7d4ac59663",
      "475b07da14204d23be8d04783f622d56",
      "54be7a1762f94831b36e253fc998bd36",
      "22c1584328674e86a35fb5aed616e6cd",
      "58488c82b68b4d8fadc43724d56c661a"
     ]
    },
    "id": "-JCKr0_n6jjb",
    "outputId": "71371e17-4cb0-4dd1-989c-5c569226b29c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading ColBERTRanker model colbert-ir/colbertv2.0\n",
      "No device set\n",
      "Using device cpu\n",
      "No dtype set\n",
      "Device set to `cpu`, setting dtype to `float32`\n",
      "Using dtype torch.float32\n",
      "Loading model colbert-ir/colbertv2.0, this might take a while...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: \n",
      "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
      "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
      "You will be able to reuse this secret in all of your notebooks.\n",
      "Please note that authentication is recommended but still optional to access public models or datasets.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a3d72b89c337400da5899d07bdc2d766",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/405 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ff434191648d4962b1dd9d537b13e5b9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8bb5cc8a235d4b628a02fd5b8121c62c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "af979629e4b84ebaa32adf0d5395e2af",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/112 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6c8dc5cadb674a33b37411aa1695bc7c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/743 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0562b3dc436c44a8b6a02e45936f2435",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/438M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linear Dim set to: {linear_dim} for downcasting\n"
     ]
    }
   ],
   "source": [
    "from lancedb.rerankers import ColbertReranker\n",
    "\n",
    "reranker_colbert = ColbertReranker()\n",
    "results_colbert = (\n",
    "    table.search(\"technologies and business models\", query_type=\"hybrid\")\n",
    "    .rerank(reranker=reranker_colbert)\n",
    "    .to_pandas()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 175
    },
    "id": "sKpzmXlhkRxh",
    "outputId": "c8ee2892-ae9d-4274-a7dd-d285ec5476dd"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "summary": "{\n  \"name\": \"results_colbert\",\n  \"rows\": 4,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"33 Dynamics products and cloud services revenue growth  Revenue from Dynamics products and cloud services, \\nincluding Dynamics 365, comprising a set of intelligent, \\ncloud -based applications across ERP, CRM, Customer \\nInsights, Power Apps, and Power Automate; and on -\\npremises ERP and CRM applications  \\n    \\nLinkedIn revenue growth  Revenue from LinkedIn, including Talent Solutions, \\nMarketing Solutions, Premium Subscriptions, and Sales \\nSolutions  \\n    \\nServer products and cloud services revenue growth  Revenue from Server products and cloud services, \\nincluding Azure and other cloud services; SQL Server, \\nWindows Server, Visual Studio, System Center, and \\nrelated Client Access Licenses (\\u201cCALs\\u201d); and Nuance and \\nGitHub  \\nMore Personal Computing  \\nMetrics related to our More Personal Computing segment assess the performance of key lines of business within this \\nsegment. These metrics provide strategic product insights which allow us to assess the performance across our \\ncommercial and consumer businesses. As we have diversity of target audiences and sales motions within the Windows \\nbusiness, we monitor metrics that are reflective of those varying motions.  \\n  \\nWindows OEM revenue growth  Revenue from sales of Windows Pro and non -Pro licenses sold \\nthrough the OEM channel  \\n    \\nWindows Commercial products and cloud \\nservices revenue growth  Revenue from Windows Commercial products and cloud services, \\ncomprising volume licensing of the Windows operating system, \\nWindows cloud services, and other Windows commercial offerings  \\n    \\nSurface revenue growth  Revenue from Surface devices and accessories  \\n    \\nXbox content and services revenue growth  Revenue from Xbox content and services, comprising first - and third -\\nparty content (including games and in -game content), Xbox Game \\nPass and other subscriptions, Xbox Cloud Gaming, third -party disc \\nroyalties, advertising, and other cloud services  \\n    \\nSearch and news advertising revenue, \\nexcluding TAC, growth  Revenue from search and news advertising excluding traffic \\nacquisition costs (\\u201cTAC\\u201d) paid to Bing Ads network publishers and \\nnews partners  \\nSUMMARY RESULTS OF OPERATIONS  \\n  \\n(In millions,  except  percentages  and per share  amounts)  2022  2021  Percentage  \\nChange    \\n        \\nRevenue  $  198,270   $  168,088    18%  \\nGross margin   135,620    115,856    17%  \\nOperating income   83,383    69,916    19%  \\nNet income   72,738    61,271    19%  \\nDiluted earnings per share   9.65   8.05   20%  \\n        \\nAdjusted net income (non -GAAP)   69,447    60,651    15%  \\nAdjusted diluted earnings per share (non -GAAP)   9.21   7.97   16%    \\nAdjusted net income and adjusted diluted earnings per share (\\u201cEPS\\u201d) are non -GAAP financial measures which exclude \\nthe net  income tax benefit related to transfer of intangible properties in the first quarter of fiscal year 2022 and the\",\n          \"34 net income tax benefit related to an India Supreme Court decision on withholding taxes in the third quarter of fiscal year \\n2021. Refer to the Non -GAAP Financial Measures section below for a reconciliation of our financial results reported in \\naccordance with GAAP to non -GAAP financial results. See Note 12 \\u2013 Income Taxes of the Notes to Financial Statements \\nin our fiscal year 2022 Form 10 -K for further discussion.  \\nFiscal Year 2022 Compared with Fiscal Year 2021  \\nRevenue increased $30.2  billion or 18% driven by growth across each of our segments. Intelligent Cloud revenue \\nincreased driven by Azure and other cloud services. Productivity and Business Processes revenue increased driven by \\nOffice 365 Commercial and LinkedIn. More Personal Computing revenue increased driven by Search and news \\nadvertising and Windows.  \\nCost of revenue increased $10.4  billion or 20% driven by growth in Microsoft Cloud.  \\nGross margin increased $19.8  billion or 17% driven by growth across each of our segments.  \\n\\u2022 Gross margin percentage decreased slightly. Excluding the impact of the fiscal year 2021 change in \\naccounting estimate for the useful lives of our server and network equipment, gross margin percentage \\nincreased 1 point driven by improvement in Productivity and Business Processes.  \\n\\u2022 Microsoft Cloud gross margin percentage decreased slightly to 70%. Excluding the impact of the change in \\naccounting estimate, Microsoft Cloud gross margin percentage increased 3 points driven by improvement \\nacross our cloud services, offset in part by sales mix shift to Azure and other cloud services.  \\nOperating expenses increased $6.3  billion or 14% driven by investments in cloud engineering, LinkedIn, Gaming, and \\ncommercial sales.  \\nKey changes in operating expenses were:  \\n\\u2022 Research and development expenses increased $3.8  billion or 18% driven by investments in cloud \\nengineering, Gaming, and LinkedIn.  \\n\\u2022 Sales and marketing expenses increased $1.7  billion or 8% driven by investments in commercial sales and \\nLinkedIn. Sales and marketing included a favorable foreign currency impact of 2%.  \\n\\u2022 General and administrative expenses increased $793  million or 16% driven by investments in corporate \\nfunctions.  \\nOperating income increased $13.5  billion or 19% driven by growth across each of our segments.  \\nCurrent year net income and diluted EPS were positively impacted by the net tax benefit related to the transfer of \\nintangible properties, which resulted in an increase to net income and diluted EPS of $3.3  billion and $0.44, respectively. \\nPrior year net income and diluted EPS were positively impacted by the net tax benefit related to the India Supreme Court \\ndecision on withholding taxes, which resulted in an increase to net income and diluted EPS of $620  million and $0.08, \\nrespectively.  \\nGross margin and operating income both included an unfavorable foreign currency impact of 2%.  \\nSEGMENT RESULTS OF OPERATIONS  \\n  \\n(In millions, except percentages)  2022  2021  Percentage  \\nChange    \\n        \\nRevenue        \\n        \\nProductivity and Business Processes  $ 63,364   $ 53,915    18%  \\nIntelligent Cloud   75,251    60,080    25%  \\nMore Personal Computing   59,655    54,093    10%        \\nTotal  $   198,270   $  168,088    18%          \\n        \\nOperating Income        \\n        \\nProductivity and Business Processes  $ 29,687   $ 24,351    22%  \\nIntelligent Cloud   32,721    26,126    25%  \\nMore Personal Computing   20,975    19,439    8%        \\nTotal  $ 83,383   $ 69,916    19%\",\n          \"31   \\nIndustry Trends  \\nOur industry is dynamic and highly competitive, with frequent changes in both technologies and business models. Each \\nindustry shift is an opportunity to conceive new products, new technologies, or new ideas that can further transform the \\nindustry and our business. At Microsoft, we push the boundaries of what is possible through a broad range of research \\nand development activities that seek to identify and address the changing demands of customers and users, industry \\ntrends, and competitive forces.  \\nEconomic Conditions, Challenges, and Risks  \\nThe markets for software, devices, and cloud -based services are dynamic and highly competitive. Our competitors are \\ndeveloping new software and devices, while also deploying competing cloud -based services for consumers and \\nbusinesses. The devices and form factors customers prefer evolve rapidly, and influence how users access services in \\nthe cloud, and in some cases, the user\\u2019s choice of which suite of cloud -based services to use. We must continue to evolve \\nand adapt over an extended time in pace with this changing environment. The investments we are making in infrastructure \\nand devices will continue to increase our operating costs and may decrease our operating margins.  \\nOur success is highly dependent on our ability to attract and retain qualified employees. We hire a mix of university and \\nindustry talent worldwide. We compete for talented individuals globally by offering an exceptional working environment, \\nbroad customer reach, scale in resources, the ability to grow one\\u2019s career across many different products and businesses, \\nand competitive compensation and benefits. Aggregate demand for our software, services, and devices is correlated to \\nglobal macroeconomic and geopolitical factors, which remain dynamic.  \\nOur devices are primarily manufactured by third -party contract manufacturers, some of which contain certain components \\nfor which there are very few qualified suppliers. For these components, we have limited near -term flexibility to use other \\nmanufacturers if a current vendor becomes unavailable or is unable to meet our requirements. Extended disruptions at \\nthese suppliers and/or manufacturers could lead to a similar disruption in our ability to manufacture devices on time to \\nmeet consumer demand.  \\nOur international operations provide a significant portion of our total revenue and expenses. Many of these revenue and \\nexpenses are denominated in currencies other than the U.S. dollar. As a result, changes in foreign exchange rates may \\nsignificantly affect revenue and expenses. Fluctuations in the U.S. dollar relative to certain foreign currencies did not hav e \\na material impact on reported revenue or expenses from our international operations in fiscal year 2022.  \\nRefer to Risk Factors in our fiscal year 2022 Form 10 -K for a discussion of these factors and other risks.  \\nSeasonality  \\nOur revenue fluctuates quarterly and is generally higher in the second and fourth quarters of our fiscal year. Second \\nquarter revenue is driven by corporate year -end spending trends in our major markets and holiday season spending by \\nconsumers, and fourth quarter revenue is driven by the volume of multi -year on -premises contracts executed during the \\nperiod.  \\nReportable Segments  \\nWe report our financial performance based on the following segments: Productivity and Business Processes, Intelligent \\nCloud, and More Personal Computing. The segment amounts included in MD&A are presented on a basis consistent with \\nour internal management reporting. Additional information on our reportable segments is contained in Note 19 \\u2013 Segment \\nInformation and Geographic Data of the Notes to Financial Statements in our fiscal year 2022 Form 10 -K.  \\nMetrics  \\nWe use metrics in assessing the performance of our business and to make informed decisions regarding the allocation of\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"vector\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"_relevance_score\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          0.705480694770813,\n          0.44533759355545044,\n          1.245123267173767\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
       "type": "dataframe",
       "variable_name": "results_colbert"
      },
      "text/html": [
       "\n",
       "  <div id=\"df-9842550c-fa7a-4bf7-9d3b-da23e7147326\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>vector</th>\n",
       "      <th>_relevance_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>31   \\nIndustry Trends  \\nOur industry is dyna...</td>\n",
       "      <td>[0.0092269145, -0.020611297, -0.0066352366, -0...</td>\n",
       "      <td>1.245123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>33 Dynamics products and cloud services revenu...</td>\n",
       "      <td>[-0.0026319, -0.023140918, -0.009075517, -0.01...</td>\n",
       "      <td>0.705481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>35   \\nReportable Segments  \\nFiscal Year 2022...</td>\n",
       "      <td>[-0.0043231365, -0.022661446, -0.014712347, -0...</td>\n",
       "      <td>0.510181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>34 net income tax benefit related to an India ...</td>\n",
       "      <td>[-0.014394735, -0.014434795, 0.0076446994, -0....</td>\n",
       "      <td>0.445338</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9842550c-fa7a-4bf7-9d3b-da23e7147326')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-9842550c-fa7a-4bf7-9d3b-da23e7147326 button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-9842550c-fa7a-4bf7-9d3b-da23e7147326');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "\n",
       "<div id=\"df-03170c44-88be-4949-8fed-92ffc0c4ad2d\">\n",
       "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-03170c44-88be-4949-8fed-92ffc0c4ad2d')\"\n",
       "            title=\"Suggest charts\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "  </button>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "      --bg-color: #E8F0FE;\n",
       "      --fill-color: #1967D2;\n",
       "      --hover-bg-color: #E2EBFA;\n",
       "      --hover-fill-color: #174EA6;\n",
       "      --disabled-fill-color: #AAA;\n",
       "      --disabled-bg-color: #DDD;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "      --bg-color: #3B4455;\n",
       "      --fill-color: #D2E3FC;\n",
       "      --hover-bg-color: #434B5C;\n",
       "      --hover-fill-color: #FFFFFF;\n",
       "      --disabled-bg-color: #3B4455;\n",
       "      --disabled-fill-color: #666;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart {\n",
       "    background-color: var(--bg-color);\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: var(--fill-color);\n",
       "    height: 32px;\n",
       "    padding: 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: var(--hover-bg-color);\n",
       "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: var(--button-hover-fill-color);\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart-complete:disabled,\n",
       "  .colab-df-quickchart-complete:disabled:hover {\n",
       "    background-color: var(--disabled-bg-color);\n",
       "    fill: var(--disabled-fill-color);\n",
       "    box-shadow: none;\n",
       "  }\n",
       "\n",
       "  .colab-df-spinner {\n",
       "    border: 2px solid var(--fill-color);\n",
       "    border-color: transparent;\n",
       "    border-bottom-color: var(--fill-color);\n",
       "    animation:\n",
       "      spin 1s steps(1) infinite;\n",
       "  }\n",
       "\n",
       "  @keyframes spin {\n",
       "    0% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "      border-left-color: var(--fill-color);\n",
       "    }\n",
       "    20% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    30% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    40% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    60% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    80% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "    90% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "  }\n",
       "</style>\n",
       "\n",
       "  <script>\n",
       "    async function quickchart(key) {\n",
       "      const quickchartButtonEl =\n",
       "        document.querySelector('#' + key + ' button');\n",
       "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
       "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
       "      try {\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      } catch (error) {\n",
       "        console.error('Error during call to suggestCharts:', error);\n",
       "      }\n",
       "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
       "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
       "    }\n",
       "    (() => {\n",
       "      let quickchartButtonEl =\n",
       "        document.querySelector('#df-03170c44-88be-4949-8fed-92ffc0c4ad2d button');\n",
       "      quickchartButtonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "    })();\n",
       "  </script>\n",
       "</div>\n",
       "\n",
       "  <div id=\"id_70eb0000-47d3-4c16-ae0a-19c5b062835e\">\n",
       "    <style>\n",
       "      .colab-df-generate {\n",
       "        background-color: #E8F0FE;\n",
       "        border: none;\n",
       "        border-radius: 50%;\n",
       "        cursor: pointer;\n",
       "        display: none;\n",
       "        fill: #1967D2;\n",
       "        height: 32px;\n",
       "        padding: 0 0 0 0;\n",
       "        width: 32px;\n",
       "      }\n",
       "\n",
       "      .colab-df-generate:hover {\n",
       "        background-color: #E2EBFA;\n",
       "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "        fill: #174EA6;\n",
       "      }\n",
       "\n",
       "      [theme=dark] .colab-df-generate {\n",
       "        background-color: #3B4455;\n",
       "        fill: #D2E3FC;\n",
       "      }\n",
       "\n",
       "      [theme=dark] .colab-df-generate:hover {\n",
       "        background-color: #434B5C;\n",
       "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "        fill: #FFFFFF;\n",
       "      }\n",
       "    </style>\n",
       "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('results_colbert')\"\n",
       "            title=\"Generate code using this dataframe.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "    <script>\n",
       "      (() => {\n",
       "      const buttonEl =\n",
       "        document.querySelector('#id_70eb0000-47d3-4c16-ae0a-19c5b062835e button.colab-df-generate');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      buttonEl.onclick = () => {\n",
       "        google.colab.notebook.generateWithVariable('results_colbert');\n",
       "      }\n",
       "      })();\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "                                                text  \\\n",
       "0  31   \\nIndustry Trends  \\nOur industry is dyna...   \n",
       "1  33 Dynamics products and cloud services revenu...   \n",
       "2  35   \\nReportable Segments  \\nFiscal Year 2022...   \n",
       "3  34 net income tax benefit related to an India ...   \n",
       "\n",
       "                                              vector  _relevance_score  \n",
       "0  [0.0092269145, -0.020611297, -0.0066352366, -0...          1.245123  \n",
       "1  [-0.0026319, -0.023140918, -0.009075517, -0.01...          0.705481  \n",
       "2  [-0.0043231365, -0.022661446, -0.014712347, -0...          0.510181  \n",
       "3  [-0.014394735, -0.014434795, 0.0076446994, -0....          0.445338  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_colbert"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "64d3D-HyRobi"
   },
   "source": [
    "Lets try the CrossEncoder for getting relavant docus from it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eJS_IdH5fZUu"
   },
   "source": [
    "## cross-encoder reranker model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hn9bxtkQeQBf"
   },
   "source": [
    "Re-ranking is a way to order results and score\n",
    "them according to their relevancy to a particular\n",
    "query.\n",
    "So let's take a look at how this works underneath.\n",
    "In re-ranking, after you retrieve results for\n",
    "a particular query, you pass these results along with\n",
    "your query to a re-ranking model. This allows you to re-rank the output\n",
    "so the most relevant results have the highest\n",
    "rank. Another way to think about this is\n",
    "your re-ranking model scores each of the results conditioned on the\n",
    "query, and those with the highest score are the most\n",
    "relevant.\n",
    "Then you can just select the top ranking\n",
    "results as the most relevant to your particular query.\n",
    "So let's take a look at how to do this in practice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 416,
     "referenced_widgets": [
      "d4b2351747314635b9eb43444c8de8e0",
      "bdaa26e75a7e4a30a11f6d66d5e9f55e",
      "1e4cdcc4f4cd4280811ac9793adc855c",
      "0555e66d46824ce8997eaa334cf80672",
      "e21a10fadc9247b99ed9fa2b6a19fe13",
      "09f02139860f478fb3831f13a6d60e8f",
      "4af4310679de42f5b0cbcc3157b122a2",
      "4c4e8d3c244d485db97472e91955cc88",
      "c18a0ca5553f46cda97bb20e95d6b1c2",
      "8960f692ece949d2b54d2181b4ca0725",
      "b3d3092adbd346999073fb3ba9664aa8",
      "09328fb6c0934ebeaee9a3b9df358e9d",
      "a66f26309d0c4d838a107681b7dc81ae",
      "f77fc0faa0c8430abc0f4f6afcbbfc3c",
      "01d478ad3fda40608780f6ad82bb4191",
      "5ab2d942d1e749c092595695a10529c3",
      "93f712bbe2a44aa18e5673601d74c98d",
      "9c3a9453e40e47c199c05443ef433c1b",
      "c6daff3dba4d4e289449e5e5e0394484",
      "eef2b395e8964342b808b96d9962555e",
      "6cd0b73ea7474afbbc206cd1007d0a75",
      "02ea4203027547d090e2150f0bd0a113",
      "f5d7bf9f8ae94584a2ad2450349f9148",
      "bcd9bc8447b14ab2ab65107c909c50b4",
      "60517d2171484e80a7f2364ad7123c7c",
      "2ae3fd9a85b34f01a41a6fc4c1fd5637",
      "b6c3c824fd1d498fae2371519cad30db",
      "3d4dc76cf6c94699b8e2756c04c952c2",
      "349f108fa0a040bab9c1c4feed85d5d5",
      "3b818834417e4b598746e9bf1328a1c3",
      "0df4bdfb09584534bc94371fde821bbc",
      "a88df049ef1641a4abd0c5131a0c422d",
      "a8c7f824c7474d8284b862b4f6f088af",
      "3a51eb69919346f6922611098dba6d54",
      "1d682c1b892f4c95969d27e317185c65",
      "cf5d9ac303894fd6a32f90020af0a238",
      "67315caa544c48caadebe84ce2801f3d",
      "054ff9eb9b3b44d892bd290ddf897f6d",
      "a93d71aea40f4fb99253913053eecfa4",
      "3cd8305cbfc3475789fdbed62f7c2b3a",
      "65e8fa4cfcb0499d9f41b05207786ad0",
      "2cbc6be96ff94e9c841257e3209b9588",
      "aab8600c8b68459cb000a2fd988d9b8e",
      "52722f3097a6492ba9d0d993b9a007b2",
      "d7acb86e9dda4dce9b78664dc53fe677",
      "2ddc6559aa6b480fbb62d2a6dd8f3927",
      "45fe6f95c51a420890fa7ba83bb6271e",
      "78d20cba9e024ad3a01f0a5230ea121f",
      "70bc85141581488ba2609738cf5a243e",
      "116d8985dc4348dc9acc833b419022b3",
      "e87010c8525f49fb9d0ec3dbbc3dda5a",
      "fd829f2c25204632a6c57cec93e6a4c7",
      "12f8dc61a70d43a7b5065fb2697022cb",
      "af266653d1d246b19e76200aa08aa315",
      "e3f98136641e4df1bb5eb5bb5717088f"
     ]
    },
    "id": "Bhg8NKLzk716",
    "outputId": "e930485c-0921-4910-9cb2-931469055c8b"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d4b2351747314635b9eb43444c8de8e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/794 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "09328fb6c0934ebeaee9a3b9df358e9d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "pytorch_model.bin:   0%|          | 0.00/90.9M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f5d7bf9f8ae94584a2ad2450349f9148",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/316 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3a51eb69919346f6922611098dba6d54",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d7acb86e9dda4dce9b78664dc53fe677",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/112 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores:\n",
      "-4.676173\n",
      "-4.906811\n",
      "-9.046174\n",
      "-3.4381006\n",
      "New Ordering:\n",
      "4\n",
      "1\n",
      "2\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# cross encoder reranker\n",
    "from sentence_transformers import CrossEncoder\n",
    "\n",
    "cross_encoder = CrossEncoder(\"cross-encoder/ms-marco-MiniLM-L-6-v2\")\n",
    "\n",
    "# Extract text content from Document objects and convert to strings\n",
    "document_texts = [doc.page_content for doc in retrieved_documents]\n",
    "query_text = (\n",
    "    \"What were the most important factors that contributed to increases in revenue?\"\n",
    ")\n",
    "# Create pairs as strings\n",
    "pairs = [[query_text, doc_text] for doc_text in document_texts]\n",
    "# Predict scores for pairs\n",
    "scores = cross_encoder.predict(pairs)\n",
    "# Print scores\n",
    "print(\"Scores:\")\n",
    "for score in scores:\n",
    "    print(score)\n",
    "\n",
    "\n",
    "print(\"New Ordering:\")\n",
    "for o in np.argsort(scores)[::-1]:\n",
    "    print(o + 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "To_3AcMjNkuc",
    "outputId": "22ca2f7b-9186-41ab-d3d8-c2d785e8c1c3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['What were the most important factors that contributed to increases in revenue?',\n",
       "  '35   \\nReportable Segments  \\nFiscal Year 2022 Compared with Fiscal Year 2021  \\nProductivity and Business Processes  \\nRevenue increased $9.4  billion or 18%.  \\n• Office Commercial products and cloud services revenue increased $4.4  billion or 13%. Office 365 \\nCommercial revenue grew 18% driven by seat growth of 14%, with continued momentum in small and \\nmedium business and frontline worker offerings, as well as growth in revenue per user. Office Commercial \\nproducts revenue declined 22% driven by continued customer shift to cloud offerings.  \\n• Office Consumer products and cloud services revenue increased $641  million or 11% driven by Microsoft 365 \\nConsumer subscription revenue. Microsoft 365 Consumer subscribers grew 15% to 59.7  million.  \\n• LinkedIn revenue increased $3.5  billion or 34% driven by a strong job market in our Talent Solutions business \\nand advertising demand in our Marketing Solutions business.  \\n• Dynamics products and cloud services revenue increased 25% driven by Dynamics 365 growth of 39%.  \\nOperating income increased $5.3  billion or 22%.  \\n• Gross margin increased $7.3  billion or 17% driven by growth in Office 365 Commercial and LinkedIn. Gross \\nmargin percentage was relatively unchanged. Excluding the impact of the change in accounting estimate, \\ngross margin percentage increased 2 points driven by improvement across all cloud services.  \\n• Operating expenses increased $2.0  billion or 11% driven by investments in LinkedIn and cloud engineering.  \\nGross margin and operating income both included an unfavorable foreign currency impact of 2%.  \\nIntelligent Cloud  \\nRevenue increased $15.2  billion or 25%.  \\n• Server products and cloud services revenue increased $14.7  billion or 28% driven by Azure and other cloud \\nservices. Azure and other cloud services revenue grew 45% driven by growth in our consumption -based \\nservices. Server products revenue increased 5% driven by hybrid solutions, including Windows Server and \\nSQL Server running in multi -cloud environments.  \\n• Enterprise Services revenue increased $464  million or 7% driven by growth in Enterprise Support Services.  \\nOperating income increased $6.6  billion or 25%.  \\n• Gross margin increased $9.4  billion or 22% driven by growth in Azure and other cloud services. Gross margin \\npercentage decreased. Excluding the impact of the change in accounting estimate, gross margin percentage \\nwas relatively unchanged driven by improvement in Azure and other cloud services, offset in part by sales \\nmix shift to Azure and other cloud services.  \\n• Operating expenses increased $2.8  billion or 16% driven by investments in Azure and other cloud services.  \\nRevenue and operating income included an unfavorable foreign currency impact of 2% and 3%, respectively.  \\nMore Personal Computing  \\nRevenue increased $5.6  billion or 10%.  \\n• Windows revenue increased $2.3  billion or 10% driven by growth in Windows OEM and Windows \\nCommercial. Windows OEM revenue increased 11% driven by continued strength in the commercial PC \\nmarket, which has higher revenue per license. Windows Commercial products and cloud services revenue \\nincreased 11% driven by demand for Microsoft 365.'],\n",
       " ['What were the most important factors that contributed to increases in revenue?',\n",
       "  '31   \\nIndustry Trends  \\nOur industry is dynamic and highly competitive, with frequent changes in both technologies and business models. Each \\nindustry shift is an opportunity to conceive new products, new technologies, or new ideas that can further transform the \\nindustry and our business. At Microsoft, we push the boundaries of what is possible through a broad range of research \\nand development activities that seek to identify and address the changing demands of customers and users, industry \\ntrends, and competitive forces.  \\nEconomic Conditions, Challenges, and Risks  \\nThe markets for software, devices, and cloud -based services are dynamic and highly competitive. Our competitors are \\ndeveloping new software and devices, while also deploying competing cloud -based services for consumers and \\nbusinesses. The devices and form factors customers prefer evolve rapidly, and influence how users access services in \\nthe cloud, and in some cases, the user’s choice of which suite of cloud -based services to use. We must continue to evolve \\nand adapt over an extended time in pace with this changing environment. The investments we are making in infrastructure \\nand devices will continue to increase our operating costs and may decrease our operating margins.  \\nOur success is highly dependent on our ability to attract and retain qualified employees. We hire a mix of university and \\nindustry talent worldwide. We compete for talented individuals globally by offering an exceptional working environment, \\nbroad customer reach, scale in resources, the ability to grow one’s career across many different products and businesses, \\nand competitive compensation and benefits. Aggregate demand for our software, services, and devices is correlated to \\nglobal macroeconomic and geopolitical factors, which remain dynamic.  \\nOur devices are primarily manufactured by third -party contract manufacturers, some of which contain certain components \\nfor which there are very few qualified suppliers. For these components, we have limited near -term flexibility to use other \\nmanufacturers if a current vendor becomes unavailable or is unable to meet our requirements. Extended disruptions at \\nthese suppliers and/or manufacturers could lead to a similar disruption in our ability to manufacture devices on time to \\nmeet consumer demand.  \\nOur international operations provide a significant portion of our total revenue and expenses. Many of these revenue and \\nexpenses are denominated in currencies other than the U.S. dollar. As a result, changes in foreign exchange rates may \\nsignificantly affect revenue and expenses. Fluctuations in the U.S. dollar relative to certain foreign currencies did not hav e \\na material impact on reported revenue or expenses from our international operations in fiscal year 2022.  \\nRefer to Risk Factors in our fiscal year 2022 Form 10 -K for a discussion of these factors and other risks.  \\nSeasonality  \\nOur revenue fluctuates quarterly and is generally higher in the second and fourth quarters of our fiscal year. Second \\nquarter revenue is driven by corporate year -end spending trends in our major markets and holiday season spending by \\nconsumers, and fourth quarter revenue is driven by the volume of multi -year on -premises contracts executed during the \\nperiod.  \\nReportable Segments  \\nWe report our financial performance based on the following segments: Productivity and Business Processes, Intelligent \\nCloud, and More Personal Computing. The segment amounts included in MD&A are presented on a basis consistent with \\nour internal management reporting. Additional information on our reportable segments is contained in Note 19 – Segment \\nInformation and Geographic Data of the Notes to Financial Statements in our fiscal year 2022 Form 10 -K.  \\nMetrics  \\nWe use metrics in assessing the performance of our business and to make informed decisions regarding the allocation of'],\n",
       " ['What were the most important factors that contributed to increases in revenue?',\n",
       "  '33 Dynamics products and cloud services revenue growth  Revenue from Dynamics products and cloud services, \\nincluding Dynamics 365, comprising a set of intelligent, \\ncloud -based applications across ERP, CRM, Customer \\nInsights, Power Apps, and Power Automate; and on -\\npremises ERP and CRM applications  \\n    \\nLinkedIn revenue growth  Revenue from LinkedIn, including Talent Solutions, \\nMarketing Solutions, Premium Subscriptions, and Sales \\nSolutions  \\n    \\nServer products and cloud services revenue growth  Revenue from Server products and cloud services, \\nincluding Azure and other cloud services; SQL Server, \\nWindows Server, Visual Studio, System Center, and \\nrelated Client Access Licenses (“CALs”); and Nuance and \\nGitHub  \\nMore Personal Computing  \\nMetrics related to our More Personal Computing segment assess the performance of key lines of business within this \\nsegment. These metrics provide strategic product insights which allow us to assess the performance across our \\ncommercial and consumer businesses. As we have diversity of target audiences and sales motions within the Windows \\nbusiness, we monitor metrics that are reflective of those varying motions.  \\n  \\nWindows OEM revenue growth  Revenue from sales of Windows Pro and non -Pro licenses sold \\nthrough the OEM channel  \\n    \\nWindows Commercial products and cloud \\nservices revenue growth  Revenue from Windows Commercial products and cloud services, \\ncomprising volume licensing of the Windows operating system, \\nWindows cloud services, and other Windows commercial offerings  \\n    \\nSurface revenue growth  Revenue from Surface devices and accessories  \\n    \\nXbox content and services revenue growth  Revenue from Xbox content and services, comprising first - and third -\\nparty content (including games and in -game content), Xbox Game \\nPass and other subscriptions, Xbox Cloud Gaming, third -party disc \\nroyalties, advertising, and other cloud services  \\n    \\nSearch and news advertising revenue, \\nexcluding TAC, growth  Revenue from search and news advertising excluding traffic \\nacquisition costs (“TAC”) paid to Bing Ads network publishers and \\nnews partners  \\nSUMMARY RESULTS OF OPERATIONS  \\n  \\n(In millions,  except  percentages  and per share  amounts)  2022  2021  Percentage  \\nChange    \\n        \\nRevenue  $  198,270   $  168,088    18%  \\nGross margin   135,620    115,856    17%  \\nOperating income   83,383    69,916    19%  \\nNet income   72,738    61,271    19%  \\nDiluted earnings per share   9.65   8.05   20%  \\n        \\nAdjusted net income (non -GAAP)   69,447    60,651    15%  \\nAdjusted diluted earnings per share (non -GAAP)   9.21   7.97   16%    \\nAdjusted net income and adjusted diluted earnings per share (“EPS”) are non -GAAP financial measures which exclude \\nthe net  income tax benefit related to transfer of intangible properties in the first quarter of fiscal year 2022 and the'],\n",
       " ['What were the most important factors that contributed to increases in revenue?',\n",
       "  '34 net income tax benefit related to an India Supreme Court decision on withholding taxes in the third quarter of fiscal year \\n2021. Refer to the Non -GAAP Financial Measures section below for a reconciliation of our financial results reported in \\naccordance with GAAP to non -GAAP financial results. See Note 12 – Income Taxes of the Notes to Financial Statements \\nin our fiscal year 2022 Form 10 -K for further discussion.  \\nFiscal Year 2022 Compared with Fiscal Year 2021  \\nRevenue increased $30.2  billion or 18% driven by growth across each of our segments. Intelligent Cloud revenue \\nincreased driven by Azure and other cloud services. Productivity and Business Processes revenue increased driven by \\nOffice 365 Commercial and LinkedIn. More Personal Computing revenue increased driven by Search and news \\nadvertising and Windows.  \\nCost of revenue increased $10.4  billion or 20% driven by growth in Microsoft Cloud.  \\nGross margin increased $19.8  billion or 17% driven by growth across each of our segments.  \\n• Gross margin percentage decreased slightly. Excluding the impact of the fiscal year 2021 change in \\naccounting estimate for the useful lives of our server and network equipment, gross margin percentage \\nincreased 1 point driven by improvement in Productivity and Business Processes.  \\n• Microsoft Cloud gross margin percentage decreased slightly to 70%. Excluding the impact of the change in \\naccounting estimate, Microsoft Cloud gross margin percentage increased 3 points driven by improvement \\nacross our cloud services, offset in part by sales mix shift to Azure and other cloud services.  \\nOperating expenses increased $6.3  billion or 14% driven by investments in cloud engineering, LinkedIn, Gaming, and \\ncommercial sales.  \\nKey changes in operating expenses were:  \\n• Research and development expenses increased $3.8  billion or 18% driven by investments in cloud \\nengineering, Gaming, and LinkedIn.  \\n• Sales and marketing expenses increased $1.7  billion or 8% driven by investments in commercial sales and \\nLinkedIn. Sales and marketing included a favorable foreign currency impact of 2%.  \\n• General and administrative expenses increased $793  million or 16% driven by investments in corporate \\nfunctions.  \\nOperating income increased $13.5  billion or 19% driven by growth across each of our segments.  \\nCurrent year net income and diluted EPS were positively impacted by the net tax benefit related to the transfer of \\nintangible properties, which resulted in an increase to net income and diluted EPS of $3.3  billion and $0.44, respectively. \\nPrior year net income and diluted EPS were positively impacted by the net tax benefit related to the India Supreme Court \\ndecision on withholding taxes, which resulted in an increase to net income and diluted EPS of $620  million and $0.08, \\nrespectively.  \\nGross margin and operating income both included an unfavorable foreign currency impact of 2%.  \\nSEGMENT RESULTS OF OPERATIONS  \\n  \\n(In millions, except percentages)  2022  2021  Percentage  \\nChange    \\n        \\nRevenue        \\n        \\nProductivity and Business Processes  $ 63,364   $ 53,915    18%  \\nIntelligent Cloud   75,251    60,080    25%  \\nMore Personal Computing   59,655    54,093    10%        \\nTotal  $   198,270   $  168,088    18%          \\n        \\nOperating Income        \\n        \\nProductivity and Business Processes  $ 29,687   $ 24,351    22%  \\nIntelligent Cloud   32,721    26,126    25%  \\nMore Personal Computing   20,975    19,439    8%        \\nTotal  $ 83,383   $ 69,916    19%']]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fezvAUtpRaV-"
   },
   "source": [
    "Now you can use this reranked documents & pass it to llm & get relavant results. please check our other techical blogs also where we have explained other important methods of how you can improve your rag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P649oDToaGle"
   },
   "source": [
    "#FlashRanker\n",
    "\n",
    "Ultra-lite & Super-fast Python library to add re-ranking to your existing search & retrieval pipelines. It is based on SoTA cross-encoders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "QrRHDFmhNM2s",
    "outputId": "dad606e6-682b-4220-e9ac-6d9c4fad0e8b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting flashrank\n",
      "  Downloading FlashRank-0.2.9-py3-none-any.whl.metadata (12 kB)\n",
      "Requirement already satisfied: tokenizers in /usr/local/lib/python3.10/dist-packages (from flashrank) (0.19.1)\n",
      "Collecting onnxruntime (from flashrank)\n",
      "  Downloading onnxruntime-1.19.2-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (4.5 kB)\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from flashrank) (1.26.4)\n",
      "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from flashrank) (2.32.3)\n",
      "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from flashrank) (4.66.5)\n",
      "Collecting coloredlogs (from onnxruntime->flashrank)\n",
      "  Downloading coloredlogs-15.0.1-py2.py3-none-any.whl.metadata (12 kB)\n",
      "Requirement already satisfied: flatbuffers in /usr/local/lib/python3.10/dist-packages (from onnxruntime->flashrank) (24.3.25)\n",
      "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from onnxruntime->flashrank) (24.1)\n",
      "Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (from onnxruntime->flashrank) (3.20.3)\n",
      "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from onnxruntime->flashrank) (1.13.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->flashrank) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->flashrank) (3.8)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->flashrank) (2.0.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->flashrank) (2024.8.30)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in /usr/local/lib/python3.10/dist-packages (from tokenizers->flashrank) (0.24.6)\n",
      "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers->flashrank) (3.16.0)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers->flashrank) (2024.6.1)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers->flashrank) (6.0.2)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers->flashrank) (4.12.2)\n",
      "Collecting humanfriendly>=9.1 (from coloredlogs->onnxruntime->flashrank)\n",
      "  Downloading humanfriendly-10.0-py2.py3-none-any.whl.metadata (9.2 kB)\n",
      "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->onnxruntime->flashrank) (1.3.0)\n",
      "Downloading FlashRank-0.2.9-py3-none-any.whl (19 kB)\n",
      "Downloading onnxruntime-1.19.2-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (13.2 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.2/13.2 MB\u001b[0m \u001b[31m56.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m2.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.8/86.8 kB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: humanfriendly, coloredlogs, onnxruntime, flashrank\n",
      "Successfully installed coloredlogs-15.0.1 flashrank-0.2.9 humanfriendly-10.0 onnxruntime-1.19.2\n"
     ]
    }
   ],
   "source": [
    "!pip install flashrank"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4NzbLIaRZ9lo",
    "outputId": "9580c879-2a0d-4b5f-a39e-510bb720f748"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ms-marco-MiniLM-L-12-v2.zip: 100%|██████████| 21.6M/21.6M [00:00<00:00, 106MiB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'text': '35   \\nReportable Segments  \\nFiscal Year 2022 Compared with Fiscal Year 2021  \\nProductivity and Business Processes  \\nRevenue increased $9.4  billion or 18%.  \\n• Office Commercial products and cloud services revenue increased $4.4  billion or 13%. Office 365 \\nCommercial revenue grew 18% driven by seat growth of 14%, with continued momentum in small and \\nmedium business and frontline worker offerings, as well as growth in revenue per user. Office Commercial \\nproducts revenue declined 22% driven by continued customer shift to cloud offerings.  \\n• Office Consumer products and cloud services revenue increased $641  million or 11% driven by Microsoft 365 \\nConsumer subscription revenue. Microsoft 365 Consumer subscribers grew 15% to 59.7  million.  \\n• LinkedIn revenue increased $3.5  billion or 34% driven by a strong job market in our Talent Solutions business \\nand advertising demand in our Marketing Solutions business.  \\n• Dynamics products and cloud services revenue increased 25% driven by Dynamics 365 growth of 39%.  \\nOperating income increased $5.3  billion or 22%.  \\n• Gross margin increased $7.3  billion or 17% driven by growth in Office 365 Commercial and LinkedIn. Gross \\nmargin percentage was relatively unchanged. Excluding the impact of the change in accounting estimate, \\ngross margin percentage increased 2 points driven by improvement across all cloud services.  \\n• Operating expenses increased $2.0  billion or 11% driven by investments in LinkedIn and cloud engineering.  \\nGross margin and operating income both included an unfavorable foreign currency impact of 2%.  \\nIntelligent Cloud  \\nRevenue increased $15.2  billion or 25%.  \\n• Server products and cloud services revenue increased $14.7  billion or 28% driven by Azure and other cloud \\nservices. Azure and other cloud services revenue grew 45% driven by growth in our consumption -based \\nservices. Server products revenue increased 5% driven by hybrid solutions, including Windows Server and \\nSQL Server running in multi -cloud environments.  \\n• Enterprise Services revenue increased $464  million or 7% driven by growth in Enterprise Support Services.  \\nOperating income increased $6.6  billion or 25%.  \\n• Gross margin increased $9.4  billion or 22% driven by growth in Azure and other cloud services. Gross margin \\npercentage decreased. Excluding the impact of the change in accounting estimate, gross margin percentage \\nwas relatively unchanged driven by improvement in Azure and other cloud services, offset in part by sales \\nmix shift to Azure and other cloud services.  \\n• Operating expenses increased $2.8  billion or 16% driven by investments in Azure and other cloud services.  \\nRevenue and operating income included an unfavorable foreign currency impact of 2% and 3%, respectively.  \\nMore Personal Computing  \\nRevenue increased $5.6  billion or 10%.  \\n• Windows revenue increased $2.3  billion or 10% driven by growth in Windows OEM and Windows \\nCommercial. Windows OEM revenue increased 11% driven by continued strength in the commercial PC \\nmarket, which has higher revenue per license. Windows Commercial products and cloud services revenue \\nincreased 11% driven by demand for Microsoft 365.', 'score': 0.19030862}, {'text': '34 net income tax benefit related to an India Supreme Court decision on withholding taxes in the third quarter of fiscal year \\n2021. Refer to the Non -GAAP Financial Measures section below for a reconciliation of our financial results reported in \\naccordance with GAAP to non -GAAP financial results. See Note 12 – Income Taxes of the Notes to Financial Statements \\nin our fiscal year 2022 Form 10 -K for further discussion.  \\nFiscal Year 2022 Compared with Fiscal Year 2021  \\nRevenue increased $30.2  billion or 18% driven by growth across each of our segments. Intelligent Cloud revenue \\nincreased driven by Azure and other cloud services. Productivity and Business Processes revenue increased driven by \\nOffice 365 Commercial and LinkedIn. More Personal Computing revenue increased driven by Search and news \\nadvertising and Windows.  \\nCost of revenue increased $10.4  billion or 20% driven by growth in Microsoft Cloud.  \\nGross margin increased $19.8  billion or 17% driven by growth across each of our segments.  \\n• Gross margin percentage decreased slightly. Excluding the impact of the fiscal year 2021 change in \\naccounting estimate for the useful lives of our server and network equipment, gross margin percentage \\nincreased 1 point driven by improvement in Productivity and Business Processes.  \\n• Microsoft Cloud gross margin percentage decreased slightly to 70%. Excluding the impact of the change in \\naccounting estimate, Microsoft Cloud gross margin percentage increased 3 points driven by improvement \\nacross our cloud services, offset in part by sales mix shift to Azure and other cloud services.  \\nOperating expenses increased $6.3  billion or 14% driven by investments in cloud engineering, LinkedIn, Gaming, and \\ncommercial sales.  \\nKey changes in operating expenses were:  \\n• Research and development expenses increased $3.8  billion or 18% driven by investments in cloud \\nengineering, Gaming, and LinkedIn.  \\n• Sales and marketing expenses increased $1.7  billion or 8% driven by investments in commercial sales and \\nLinkedIn. Sales and marketing included a favorable foreign currency impact of 2%.  \\n• General and administrative expenses increased $793  million or 16% driven by investments in corporate \\nfunctions.  \\nOperating income increased $13.5  billion or 19% driven by growth across each of our segments.  \\nCurrent year net income and diluted EPS were positively impacted by the net tax benefit related to the transfer of \\nintangible properties, which resulted in an increase to net income and diluted EPS of $3.3  billion and $0.44, respectively. \\nPrior year net income and diluted EPS were positively impacted by the net tax benefit related to the India Supreme Court \\ndecision on withholding taxes, which resulted in an increase to net income and diluted EPS of $620  million and $0.08, \\nrespectively.  \\nGross margin and operating income both included an unfavorable foreign currency impact of 2%.  \\nSEGMENT RESULTS OF OPERATIONS  \\n  \\n(In millions, except percentages)  2022  2021  Percentage  \\nChange    \\n        \\nRevenue        \\n        \\nProductivity and Business Processes  $ 63,364   $ 53,915    18%  \\nIntelligent Cloud   75,251    60,080    25%  \\nMore Personal Computing   59,655    54,093    10%        \\nTotal  $   198,270   $  168,088    18%          \\n        \\nOperating Income        \\n        \\nProductivity and Business Processes  $ 29,687   $ 24,351    22%  \\nIntelligent Cloud   32,721    26,126    25%  \\nMore Personal Computing   20,975    19,439    8%        \\nTotal  $ 83,383   $ 69,916    19%', 'score': 0.04015541}, {'text': '31   \\nIndustry Trends  \\nOur industry is dynamic and highly competitive, with frequent changes in both technologies and business models. Each \\nindustry shift is an opportunity to conceive new products, new technologies, or new ideas that can further transform the \\nindustry and our business. At Microsoft, we push the boundaries of what is possible through a broad range of research \\nand development activities that seek to identify and address the changing demands of customers and users, industry \\ntrends, and competitive forces.  \\nEconomic Conditions, Challenges, and Risks  \\nThe markets for software, devices, and cloud -based services are dynamic and highly competitive. Our competitors are \\ndeveloping new software and devices, while also deploying competing cloud -based services for consumers and \\nbusinesses. The devices and form factors customers prefer evolve rapidly, and influence how users access services in \\nthe cloud, and in some cases, the user’s choice of which suite of cloud -based services to use. We must continue to evolve \\nand adapt over an extended time in pace with this changing environment. The investments we are making in infrastructure \\nand devices will continue to increase our operating costs and may decrease our operating margins.  \\nOur success is highly dependent on our ability to attract and retain qualified employees. We hire a mix of university and \\nindustry talent worldwide. We compete for talented individuals globally by offering an exceptional working environment, \\nbroad customer reach, scale in resources, the ability to grow one’s career across many different products and businesses, \\nand competitive compensation and benefits. Aggregate demand for our software, services, and devices is correlated to \\nglobal macroeconomic and geopolitical factors, which remain dynamic.  \\nOur devices are primarily manufactured by third -party contract manufacturers, some of which contain certain components \\nfor which there are very few qualified suppliers. For these components, we have limited near -term flexibility to use other \\nmanufacturers if a current vendor becomes unavailable or is unable to meet our requirements. Extended disruptions at \\nthese suppliers and/or manufacturers could lead to a similar disruption in our ability to manufacture devices on time to \\nmeet consumer demand.  \\nOur international operations provide a significant portion of our total revenue and expenses. Many of these revenue and \\nexpenses are denominated in currencies other than the U.S. dollar. As a result, changes in foreign exchange rates may \\nsignificantly affect revenue and expenses. Fluctuations in the U.S. dollar relative to certain foreign currencies did not hav e \\na material impact on reported revenue or expenses from our international operations in fiscal year 2022.  \\nRefer to Risk Factors in our fiscal year 2022 Form 10 -K for a discussion of these factors and other risks.  \\nSeasonality  \\nOur revenue fluctuates quarterly and is generally higher in the second and fourth quarters of our fiscal year. Second \\nquarter revenue is driven by corporate year -end spending trends in our major markets and holiday season spending by \\nconsumers, and fourth quarter revenue is driven by the volume of multi -year on -premises contracts executed during the \\nperiod.  \\nReportable Segments  \\nWe report our financial performance based on the following segments: Productivity and Business Processes, Intelligent \\nCloud, and More Personal Computing. The segment amounts included in MD&A are presented on a basis consistent with \\nour internal management reporting. Additional information on our reportable segments is contained in Note 19 – Segment \\nInformation and Geographic Data of the Notes to Financial Statements in our fiscal year 2022 Form 10 -K.  \\nMetrics  \\nWe use metrics in assessing the performance of our business and to make informed decisions regarding the allocation of', 'score': 0.0031708966}, {'text': '33 Dynamics products and cloud services revenue growth  Revenue from Dynamics products and cloud services, \\nincluding Dynamics 365, comprising a set of intelligent, \\ncloud -based applications across ERP, CRM, Customer \\nInsights, Power Apps, and Power Automate; and on -\\npremises ERP and CRM applications  \\n    \\nLinkedIn revenue growth  Revenue from LinkedIn, including Talent Solutions, \\nMarketing Solutions, Premium Subscriptions, and Sales \\nSolutions  \\n    \\nServer products and cloud services revenue growth  Revenue from Server products and cloud services, \\nincluding Azure and other cloud services; SQL Server, \\nWindows Server, Visual Studio, System Center, and \\nrelated Client Access Licenses (“CALs”); and Nuance and \\nGitHub  \\nMore Personal Computing  \\nMetrics related to our More Personal Computing segment assess the performance of key lines of business within this \\nsegment. These metrics provide strategic product insights which allow us to assess the performance across our \\ncommercial and consumer businesses. As we have diversity of target audiences and sales motions within the Windows \\nbusiness, we monitor metrics that are reflective of those varying motions.  \\n  \\nWindows OEM revenue growth  Revenue from sales of Windows Pro and non -Pro licenses sold \\nthrough the OEM channel  \\n    \\nWindows Commercial products and cloud \\nservices revenue growth  Revenue from Windows Commercial products and cloud services, \\ncomprising volume licensing of the Windows operating system, \\nWindows cloud services, and other Windows commercial offerings  \\n    \\nSurface revenue growth  Revenue from Surface devices and accessories  \\n    \\nXbox content and services revenue growth  Revenue from Xbox content and services, comprising first - and third -\\nparty content (including games and in -game content), Xbox Game \\nPass and other subscriptions, Xbox Cloud Gaming, third -party disc \\nroyalties, advertising, and other cloud services  \\n    \\nSearch and news advertising revenue, \\nexcluding TAC, growth  Revenue from search and news advertising excluding traffic \\nacquisition costs (“TAC”) paid to Bing Ads network publishers and \\nnews partners  \\nSUMMARY RESULTS OF OPERATIONS  \\n  \\n(In millions,  except  percentages  and per share  amounts)  2022  2021  Percentage  \\nChange    \\n        \\nRevenue  $  198,270   $  168,088    18%  \\nGross margin   135,620    115,856    17%  \\nOperating income   83,383    69,916    19%  \\nNet income   72,738    61,271    19%  \\nDiluted earnings per share   9.65   8.05   20%  \\n        \\nAdjusted net income (non -GAAP)   69,447    60,651    15%  \\nAdjusted diluted earnings per share (non -GAAP)   9.21   7.97   16%    \\nAdjusted net income and adjusted diluted earnings per share (“EPS”) are non -GAAP financial measures which exclude \\nthe net  income tax benefit related to transfer of intangible properties in the first quarter of fiscal year 2022 and the', 'score': 0.00047622027}]\n"
     ]
    }
   ],
   "source": [
    "# Metadata is optional, Id can be your DB ids from your retrieval stage or simple numeric indices.\n",
    "from flashrank import Ranker, RerankRequest\n",
    "\n",
    "# query = \"How to speedup LLMs?\"\n",
    "\n",
    "query = \"What were the most important factors that contributed to increases in revenue?\"\n",
    "\n",
    "ranker = Ranker(model_name=\"ms-marco-MiniLM-L-12-v2\", cache_dir=\"/opt\")\n",
    "\n",
    "rerankrequest = RerankRequest(query=query, passages=formatted_data)\n",
    "results = ranker.rerank(rerankrequest)\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bzHa6zld2wQa"
   },
   "source": [
    "**This is how you can use this methods & pass this data to llm & get better results fro your RAG applications**"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
